Statistical Techniques to Assess Publication Integrity in Groups of Randomised Trials: A Narrative Review.

Objective To describe statistical tools available for assessing publication integrity of groups of randomised controlled trials (RCTs) Study Design Narrative review Results Freely available statistical tools have been developed that compare the observed distributions of baseline variables with the expected distributions that would occur if successful randomisation occurred. For continuous variables, the tools assess baseline means, baseline p-values, and the occurrence of identical means and/or SD. For categorical variables, they assess baseline p-values, frequency counts for individual or all variables, numbers of trial participants randomised or withdrawing, and compare reported with independently calculated p-values. The tools have been used to identify publication integrity concerns in RCTs from individual groups, and performed at an acceptable level in discriminating intentionally fabricated baseline summary data from data from genuine RCTs. The tools can be used when concerns have been raised about RCT(s) from an individual/group and when the whole body of their work is being examined, when conducting systematic reviews, and could be adapted to aid screening of RCTs at journal submission. Conclusion Statistical tools are useful for the assessment of publication integrity of groups of RCTs.

Responses to concerns raised about publications don't address the concerns raised.

Recently in the Journal, Amanda Williams described her experience of raising concerns about a group of trials with "untrustworthy data". We were inspired by the work of Williams and colleagues to examine these and other trials by the same research group. Similar to Williams, we found that the patterns of reported data differed from the patterns expected to arise from valid randomisation. We also identified a high proportion of reported baseline p-values for categorial variables that differed from independently calculated p-values. We reported these findings to the affected journals but none of the concerns were addressed and no action will be taken about the majority. Despite the large number of unresolved concerns about these trials, readers will be unaware of the issues, which seems entirely unsatisfactory.

Ten Years later: Assessments of the integrity of publications from one research group with multiple retractions.

When a research group has multiple retracted publications and/or research misconduct by a member is evident, there is a risk that its other publications are unreliable, so a comprehensive assessment of the group's publications is advisable. We analyzed the comprehensiveness of assessment of the integrity of 300 publications by a research group with numerous retractions and known research misconduct, for 292 of which we raised concerns to publishers and academic institutions between 3/2013 and 2/2020. By 4/2023, 91 (30%) publications had not been assessed by either publisher or academic institution. Publishers had assessed 185 (63%) publications. The 4 academic institutions had assessed 5/36 (14%), 56/216 (26%), 30/50 (60%) and 40/66 (61%) publications. Unprompted assessments, those undertaken without our notification of concerns, occurred for 24 (8%) publications, 3 (1%) by publishers and 21 (7%) by academic institutions. Among 32 journals with ≥2 affected publications, no unprompted assessments of the remaining publication(s) occurred after notification of concerns about the index publication(s). Publishers retracted 58/84 (69%) publications which institutions also assessed and decided needed no editorial action. These analyses demonstrate the failure of publishers and institutions to comprehensively and spontaneously determine the integrity of publications in a setting of known misconduct and multiple retractions.

Increased workload without clinical benefit: Results following implementation of the ACR-TIRADS system for thyroid nodules.

OBJECTIVE: The ACR-TIRADS system for stratifying thyroid nodule malignancy risk has been widely promoted and implemented. We audited its introduction at a large public hospital in Auckland, New Zealand. DESIGN: Audit of outcomes following thyroid nodule fine needle aspiration (FNA) before/after ACR-TIRADS. PATIENTS: Individuals undergoing thyroid FNA 2017-2019. MEASUREMENTS: From medical records, we obtained details from the pre-FNA ultrasound (nodule size, TIRADS points/levels, radiologist recommendation for FNA), Bethesda (B) cytology classification, histology and post-FNA follow-up. RESULTS: Four hundred and twenty-two individuals had 564 FNAs, 163 had surgery and 54 (13%) had cancer in the primary nodule. 37/54 (69%) cancers were papillary thyroid carcinoma (median size 25 mm, 87% ≥10 mm, 61% ≥20 mm). Following ACR-TIRADS introduction, FNA recommendations increased greater than twofold, FNAs performed by 71%-83%, and the monthly rate of FNAs and operations by 60% and 40%, respectively. However, the proportion of cancers/FNA remained similar (9.9% post-TIRADS vs. 8.7% pre-TIRADS). The proportions of FNA results remained stable for B2-B4 categories, but doubled (11% vs. 5%) for B5-B6: 15 FNAs were needed to identify an additional B5/B6 lesion. TIRADS-5 nodules had a higher proportion of B5/B6 (20%) and a lower proportion of B2 (30%) than TIRADS-3 (2%, 57%, respectively) and TIRADS-4 (9%, 56%) nodules. About 5 additional cancers/year were diagnosed, but they were more often small (49% vs. 8% <2 cm, 17% vs. 0% <1 cm). CONCLUSION: ACR-TIRADS introduction increased workload (FNAs and operations), without increasing the proportion of cancers/FNA. It led to a few more cancers being diagnosed, but many were small and of uncertain clinical significance.

Distributions of baseline categorical variables were different from the expected distributions in randomized trials with integrity concerns.

BACKGROUND AND OBJECTIVES: Comparing observed and expected distributions of baseline continuous variables in randomized controlled trials (RCTs) can be used to assess publication integrity. We explored whether baseline categorical variables could also be used. METHODS: The observed and expected (binomial) distribution of all baseline categorical variables were compared in four sets of RCTs: two controls, and two with publication integrity concerns. We also compared baseline calculated and reported P-values. RESULTS: The observed and expected distributions of baseline categorical variables were similar in the control datasets, both for frequency counts (and percentages) and for between-group differences in frequency counts. However, in both sets of RCTs with publication integrity concerns, about twice as many variables as expected had between-group differences in frequency counts of one or 2, and far fewer variables than expected had between-group differences of >4 (P < 0.001 for both datasets). Furthermore, about one in six reported P-values for baseline categorial variables differed by > 0.1 from the calculated P-value in trials with publication integrity concerns. CONCLUSION: Comparing the observed and expected distributions and reported and calculated P-values of baseline categorical variables may help in the assessment of publication integrity of a body of RCTs.

Is PHARMAC's decision-making fair, cost-effective and clinically effective? Observations from the real world.

Nil.

Zoledronate Reduces Height Loss Independently of Vertebral Fracture Occurrence in a Randomized Trial in Osteopenic Older Women.

Vertebral fractures are associated with height loss, reduced quality of life, and increased mortality and are an important endpoint for osteoporosis trials. However, height loss is associated with quality of life and mortality independent of associations with fracture. We have used data from a recent 6-year trial of zoledronate in 2000 osteopenic women aged >65 years to assess the impact of the semiquantitative and quantitative components of the definition of vertebral fracture on the outcome of that trial, to determine what factors impacted on height loss and to test whether height loss can be used as a surrogate for vertebral fracture incidence. In the trial protocol, an incident vertebral fracture was defined as a change in Genant grade plus both a 20% and 4 mm decrease in a vertebral height. The addition of the quantitative criteria reduced the number of fractures detected but did not change the size of the anti-fracture effect (odds ratios of 0.49 versus 0.45) nor the width of the confidence intervals for the odds ratios. Multivariate analysis of baseline predictors of height change showed that age accelerated height loss (p < 0.0001) and zoledronate reduced it (p = 0.0001). Incident vertebral fracture increased height loss (p = 0.0005) but accounted for only 0.7% of the variance in height change, so fracture could not be reliably inferred from height loss. In women without incident vertebral fractures, height loss was still reduced by zoledronate (height change: zoledronate, -1.23; placebo -1.51 mm/yr, p < 0.0001). This likely indicates that zoledronate prevents a subtle but widespread loss of vertebral body heights not detected by vertebral morphometry. Because height loss is associated with quality of life and mortality independent of associations with fracture, it is possible that zoledronate impacts on these endpoints via its effects on vertebral body integrity. © 2022 American Society for Bone and Mineral Research (ASBMR).

Procrastination and inconsistency: Expressions of concern for publications with compromised integrity.

Expressions of concern (EoC) can reduce the adverse effects of unreliable publications by alerting readers to concerns about publication integrity while assessment is undertaken. We investigated the use of EoC for 463 publications by two research groups for which we notified concerns about publication integrity to 142 journals and 44 publishers between March 2013 and February 2020. By December 2021, 95 papers had had an EoC, and 83 were retracted without an EoC. Median times from notification of concerns to EoC (10.4mo) or retraction without EoC (13.1mo) were similar. Among the 95 EoCs, 29 (30.5%) were followed by retraction after a median of 5.4mo, none was lifted, and 66 (69.5%) remained in place after a median of 18.1mo. Publishers with >10 notified publications issued EoCs for 0-81.8% of papers: for several publishers the proportions of notified papers for which EoCs were issued varied considerably between the 2 research groups. EoCs were issued for >30% of notified publications of randomized clinical trials and letters to the editor, and <20% of other types of research. These results demonstrate inconsistent application of EoCs between and within publishers, and prolonged times to issue and resolve EoCs.

A randomized trial alerting authors, with or without coauthors or editors, that research they cited in systematic reviews and guidelines has been retracted.

Retracted clinical trials may be influential in citing systematic reviews and clinical guidelines. We assessed the influence of 27 retracted trials on systematic reviews and clinical guidelines (citing publications), then alerted authors to these retractions. Citing publications were randomized to up to three e-mails to contact author with/without up to two coauthors, with/without the editor. After one year we assessed corrective action. We included 88 citing publications; 51% (45/88) had findings likely to change if retracted trials were removed, 87% (39/45) likely substantially. 51% (44/86) of contacted citing publications replied. Including three authors rather than the contact author alone was more likely to elicit a reply (P = 0.03). Including the editor did not increase replies (P = 0.66). Whether findings were judged likely to change, and size of the likely change, had no effect on response rate or action taken. One year after e-mails were sent only nine publications had published notifications. E-Mail alerts to authors and editors are inadequate to correct the impact of retracted publications in citing systematic reviews and guidelines. Changes to bibliographic and referencing systems, and submission processes are needed. Citing publications with retracted citations should be marked until authors resolve concerns.

Diversity of invited speakers at endocrinology conferences.

OBJECTIVE: To assess diversity of gender and geographical location of institutional affiliation among invited speakers at major international endocrinology conferences. DESIGN AND METHODS: Descriptive study of characteristics of invited speakers at eight general and discipline-specific endocrinology conferences held annually in Europe and North America 2013-2019. Main outcomes were gender, geographical location of institutional affiliation and frequency of repeat presentations among invited speakers. RESULTS: Of 2375 invited speakers who gave 3522 presentations, 843 (35.5%) were women. Five hundred and ninety-four (25.0%) speakers gave >1 presentation at any conference. The proportion of women speakers declined as the number of presentations per speaker increased. Of speakers giving two and seven presentations, respectively, 36.0% and 20.0% were women. 52.9% of speakers were from institutions in North America, and 25.6% from institutions in Western Europe. Fewer than 5% of speakers were from institutions in each of Eastern Europe, Asia, South America, Africa and Oceania. The proportions of speakers who were women and from each geographical area were unchanged over 7 years. Up to one in three speakers gave >1 presentation at an individual conference (range 9.9%-32.2%). CONCLUSIONS: Women and speakers from institutions outside of North America and Western Europe are underrepresented among invited speakers at major international endocrinology conferences. Longitudinal data indicate no change in either speaker characteristic over the time period examined. These underrepresentations are more marked among speakers who give repeat presentations.

Citation of retracted publications: A challenging problem.

Scientific publications with compromised integrity should be retracted. Papers citing retracted publications might need correction if findings depend on the retracted publication. While many studies have reported on post-retraction citations, few have focused on citations made before the retraction. We investigated the citation profile for a research group with 113 published concerns regarding publication integrity (CRPI). We identified 376 of their source publications that were cited by 5577 articles, and whether the source publication had a published CRPI. Of 6926 references to a source publication in these citing articles, for 3925 (57%) the source article had a published CRPI, while for 3001 (43%) it did not. Of these 3925 references, 3688 were in citing articles published before the source article CRPI was published. 166 citing articles containing 198 references to source publications were published after the corresponding source article CRPI was published (range 1-5 such references/article; 19/166 (11%) articles had >1 reference). In summary, many articles cite retracted publications, with the majority of these references occurring before the retraction. However, very few publications assess the impact of the retracted citations, even though the findings of many might be altered, at least in part, by removal of the retracted citation.

Dietary calcium intake and change in bone mineral density in older adults: a systematic review of longitudinal cohort studies.

Many older adults do not achieve recommended intakes of calcium and there is some concern over the potential impact of this on bone health. The objective of this review was to examine evidence from cohort studies on the relationship between calcium intake and change in bone mineral density (BMD) in older adults, something not undertaken in the last two decades. Data sources included Ovid Medline, Embase, and PubMed and references from retrieved reviews and articles. The final search was performed in February 2021. We included cohort studies of calcium intake in participants aged >50 years with change in BMD over ≥1 year as an outcome. We identified 23 studies of women and 7 of men. Most studies found no association between calcium intake and change in BMD in women (71%) or men (71%). Among women, five studies reported high rates (>30% of participants) of hormone treatment or osteoporosis therapy (HT/OT) use; 80% of these studies reported a positive association between calcium intake and change in BMD, compared with 10% of studies in which HT/OT use was low. No study in women in which the mean age was >60 years reported a positive association between calcium intake and change in BMD. We conclude that calcium intake across the ranges consumed in these studies (mean intake in all but one study >500 mg/day) is not an important determinant of bone loss, particularly among women >60 years. The positive findings in studies with high rates of HT/OT use are likely to arise from confounding as a result of co-administration of calcium supplements with these medications.

Bone Mineral Density and Bone Turnover 10 Years After a Single 5 mg Dose or Two 5-Yearly Lower Doses of Zoledronate in Osteopenic Older Women: An Open-Label Extension of a Randomized Controlled Trial.

Intravenous zoledronate reduces fracture risk (5 mg at 18-month intervals) and prevents bone loss (doses of 1 to 5 mg for 3 to >5 years), but the duration of action of a single 5 mg dose and the effects of lower doses beyond 5 years are unknown. We report the second open-label extension (years 5 to 10) of a 2-year randomized, multidose, placebo-controlled, double-blinded trial. A total of 116 older women who completed 5 years of participation either continued observation without further treatment (zoledronate 5 mg and placebo at baseline) or received repeat doses of 1 or 2.5 mg zoledronate (zoledronate 1 mg and zoledronate 2.5 mg at baseline, respectively). Outcomes were spine, hip, and total body bone mineral density (BMD) and serum markers of bone turnover. After a single 5 mg dose of zoledronate, mean BMD at the lumbar spine and total hip was maintained at or above baseline levels for 9 and 10 years, respectively. The mean level of the bone resorption marker ß-C-terminal telopeptide of type I collagen (ß-CTX) was at least 25% lower than that in the placebo group for 9 years. In women administered 5-yearly doses of 2.5 mg zoledronate, mean BMD at the total hip and lumbar spine was maintained at or above baseline levels for 9 and 10 years, respectively. Redosing with 1 or 2.5 mg zoledronate at 5 years reduced bone turnover markers for 3 to 4 years. BMD increased for 3 to 4 years after redosing with 1 mg zoledronate. In the group given 5-yearly 2.5 mg zoledronate, ß-CTX was at least 20% lower than that in the placebo group for 10 years. Both a single baseline 5 mg dose of zoledronate and 5-yearly doses of 1 and 2.5 mg zoledronate prevented bone loss at hip and spine for 8 to 10 years in older postmenopausal women. Clinical trials to evaluate the effects on fracture risk of these very infrequent and lower doses of zoledronate are justified. © 2021 American Society for Bone and Mineral Research (ASBMR).

Vitamin D deficiency, supplementation and testing: have we got it right in New Zealand?

BACKGROUND: Severe prolonged vitamin D deficiency can cause rickets or osteomalacia. Both can be prevented by sunshine exposure or vitamin D supplementation. Although New Zealand guidance does not recommend vitamin D supplementation for the general population, it can be considered for individuals at risk of vitamin D deficiency. Routine measurement of 25-hydroxyvitamin D (25OHD) is also considered unnecessary. METHODS: We investigated the rates of vitamin D supplementation, rickets and osteomalacia in New Zealand, and of 25OHD results in Auckland, over the last two decades. RESULTS: Vitamin D prescriptions increased 14-fold, from 86,295/year to 1,215,507/year, between 2003 and 2019, with medication costs alone in 2019 being >$1 million. Despite these changes, the annual prevalence of hospital admissions for rickets, osteomalacia and unspecified vitamin D deficiency remained low and stable (10-20/year). 25OHD concentrations increased between 2002 and 2003 and between 2009 and 2019, and in the later time-period, 25OHD tests mainly identified individuals without vitamin D deficiency (40-50% >75nmol/L, 65-70% >50nmol/L and only 7-12.5% <25nmol/L). CONCLUSIONS: Osteomalacia and rickets persist at low rates despite widespread, increasingly costly vitamin D supplementation and testing, which largely identifies individuals without vitamin D deficiency. These results suggest that vitamin D guidance and practice in New Zealand should change.